A router with a circle jig thingy is the cats meow when doing the hole cutouts, at this point it's either live with it or if your real good and buy a circle jig and a router you could carve a big enough hole in a sheet of ply for the router to run around inside of and have it just the right size so it just skims your hole out and makes it nice and round again.